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Abstract: The aim of the paper is to present the scientific model and several methods for the expert evaluation of 
quality of learning objects (LOs) paying especial attention to LOs reusability level. The activities of eQNet Quality 
Network for a European Learning Resource Exchange (LRE) aimed to improve reusability of LOs of European 
Schoolnet’s LRE service for schools are analysed in more detail. As a pan-European service, the LRE particularly 
seeks to identify LOs that can “travel well” (i.e., reusable) across national borders and can be used in a cultural 
and linguistic context different from the one in which they were created. The primary aim is to improve the quality 
of LOs in LRE. eQNet is doing this by establishing a network consisting of researchers, policy makers, and 
practitioners (teachers) that develops and applies “travel well” quality criteria to both existing LRE content as well 
as that to be selected in future from national repositories. The vision driving the LRE is that a significant 
percentage of high quality LOs developed in different countries, in different languages and to meet the needs of 
different curricula can be re-used at European level. The main problem of all existing approaches in the area is a 
high level of the expert evaluation subjectivity. The authors analyse several scientific approaches, theories, 
methods and principles to minimise the subjectivity level in expert evaluation of LOs quality, namely: (1) multiple 
criteria decision analysis approaches for identification of quality criteria, (2) technological quality criteria 
classification principle, (c) fuzzy group decision making theory to obtain evaluation measures, (d) normalisation 
requirement for criteria weights, (e) scalarisation method and (f) trapezoidal fuzzy method for LOs quality 
optimisation. The authors show that the complex application of these approaches could significantly improve the 
quality of expert evaluation of LOs and noticeably reduce the expert evaluation subjectivity level. The paper also 
presents several examples of practical application of these approaches for LOs quality evaluation for Physics and 
Mathematics subjects. 
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1. Introduction: Evaluation of learning objects in eQNet project 

The aim of the paper is to present the scientific model and several methods for the expert evaluation 
of quality of learning objects (LOs) paying especial attention to their reusability level. This is also one 
of the main objectives of a new eQNet (eQNet 2011) project. 

eQNet is a three-year (September 2009-2012) Comenius Multilateral Network funded under the 
European Commission’s Lifelong Learning programme. The project is coordinated by European 
Schoolnet (EUN) and involves 9 Ministries of Education or agencies nominated to act of their 
behalf. The primary aim is to improve the quality of learning objects (LOs) in European Schoolnet’s 
Learning Resource Exchange - LRE (LRE 2011) which currently offers almost 130,000 LOs and 
assets from over 25 providers. As a pan-European service, the LRE particularly seeks to identify LOs 
that “travel well” (i.e., reusable) across national borders and can be used in a cultural and linguistic 
context different from the one in which they were created (eQNet 2011). 

eQNet is doing this by establishing a network consisting of researchers, policy makers and 
practitioners (teachers) that develops and applies “travel well” quality criteria to both existing LRE 
content as well as that to be selected in future from national repositories. The vision driving the LRE is 
that a significant percentage of high quality LOs developed in different countries, in different 
languages and to meet the needs of different curricula can be re-used at European level. eQNet 
provides a forum for joint reflection and co-operation related to the exchange and re-use of 
educational content and allows network members to: 

■ Better share information and expertise particularly related to “travel well” quality criteria (i.e., 
pedagogical, technical and intellectual property rights (IPR) factors); 

■ Develop new frameworks to improve the quality of LOs and metadata in both national repositories 
and the LRE, including the growing volume of user-generated content and metadata, as well as to 
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improve the multilinguality of LRE content as a result of the translation of metadata, making use, 
where appropriate, of automatic metadata translation approaches and technologies; 

■ Enable schools to participate in a Community of Practice related to the use LOs at European 
level. 

Major results will include: 

■ The development of “travel well” quality criteria to more easily identify LOs with the potential for 
cross-border use (this work package is coordinated by the Lithuanian partner, in particular by the 
author of the paper); 

■ The practical application by teachers of these criteria to >3,500 LOs in the LRE; 

■ ‘Showcases’ of the best of these LOs in a “travel well” section of the LRE portal; 

■ Where necessary, the enrichment of selected LOs with new or better metadata; 

■ A Community of Practice for teachers around these LOs (eQNet 2011). 

2. Methodology of the research 

One of the main features achieving the high LOs effectiveness and efficiency level is LOs reusability. 
The need for reusability of LOs has at least three elements (McCormick et. al. 2004, Kurilovas 2009): 

(1) Interoperability: LO is interoperable and can be used in different platforms. 

(2) Flexibility in terms of pedagogic situations: LO can fit into a variety of pedagogic situations. 

(3) Modifiability to suit a particular teacher’s or student’s needs: LO can be made more appropriate to 
a pedagogic situation by modifying it to suit a particular teacher’s or student’s needs. 

Reusability of LOs (or their ability to “travel well” between different contexts and education systems) is 
considered by the authors as a part of the overall quality of LOs. This means that any high quality LO 
has some reusability level (or potential to “travel well”), but this does not mean that any reusable LO is 
quality one. 

The main problem analysed in the paper is how to establish 

(1) a ‘proper’ set of LOs “travel well” quality evaluation criteria that should reflect the objective 
scientific principles of construction a model (criteria tree) for LOs “travel well” quality evaluation, and 

(2) ‘proper’ methods for evaluation of LOs “travel well” quality. 

According to (Oliver 2000), evaluation can be characterised as “the process by which people make 
judgements about value and worth”. In the context of learning technology this judgement process is 
complex and often controversial. Although the notion of evaluation is rooted in a relatively simple 
concept, the process of judging the value of learning technology is complex and challenging. Quality 
evaluation is defined as “the systematic examination of the extent to which an entity (part, product, 
service or organisation) is capable of meeting specified requirements” (ISO/IEC 1999). Expert 
evaluation is referred here as the multiple criteria evaluation of LOs aimed at selection of the best 
alternatives (i.e., LOs) based on score-ranking results (Kurilovas and Dagiene 2009a). 

According to (Zavadskas and Turskis 2010), there is a wide range of multiple criteria decision making 
(MCDM) problem solution techniques, varying in complexity and possible solutions. Each method has 
its own strength, weaknesses and possibilities to be applied. But, according to (Zavadskas and 
Turskis 2010), there are still no rules determining the application of multi-criteria evaluation methods 
and interpretation of the results obtained. 

If the set of decision alternatives (LOs) is assumed to be predefined, fixed and finite, then the decision 
problem is to choose the optimal alternative or, maybe, to rank them. But usually the experts have to 
deal with the problem of optimal decision in the multiple criteria situation where the objectives are 
often conflicting. In this case, an optimal decision is the one that maximises the expert’s utility. 
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These principles of identification of quality evaluation criteria have been analysed in multiple criteria 
decision analysis (MCDA) theory related research works (e.g., (Belton and Stewart 2002)). 

Evaluation of LOs quality is a typical case where the criteria are conflicting, i.e., LOs could be very 
qualitative against several criteria, and not qualitative against the other ones, and vice versa. 
Therefore, the authors propose to use MCDA approach for creation of LOs quality evaluation model. 

LOs multiple criteria evaluation method used by the authors in eQNet is referred here as the experts’ 
additive utility function represented by formula (1) below including LOs evaluation criteria, their ratings 
(values) and weights (Kurilovas and Serikoviene 2010). 

This method is well-known in the theory of optimisation methods and is named “scalarisation method 
A possible decision here could be to transform multi-criteria task into one-criterion task obtained by 
adding all criteria together with their weights. It is valid from the point of view of the optimisation 
theory, and a special theorem exists for this case (Kurilovas and Serikoviene 2010). 


Therefore, here we have the experts' additive utility function: 
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where f , (Xj) is the rating (i.e., non-fuzzy value) of the criterion / for the each of the examined LOs 


alternatives Xj. The weights here should be ‘normalised’ according to the ‘normalisation’requirement 
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According to (Zavadskas and Turskis 2010), the normalisation aims at obtaining comparable scales of 
criteria values. The major is the meaning of the utility function (1) the better LOs meet the quality 
requirements in comparison with the ideal (i.e., 100%) quality. 

3. Literature analysis and research results 

This section is aimed to apply the aforementioned scientific approaches in order: 

(1) to propose a suitable scientific model for evaluation of quality of LOs, 

(2) to propose suitable scientific methods for evaluation of quality of LOs, and 

(3) to present the experimental evaluation results using the proposed evaluation model and methods. 

3.1 Learning objects quality evaluation model 

The following principles of identification of quality evaluation criteria are relevant to all MCDA 
approaches (Belton and Stewart 2002): 

(1) Value relevance: Are the decision makers able to link the concept to their goals, thereby enabling 
them to specify preferences which relate directly to the concept? 

(2) Understandability: It is important that decision makers have a shared understanding of concepts to 
be used in an analysis. 

(3) Measurability: All MCDA implies some degree of measurement of the performance of alternatives 
against specified criteria, thus it must be possible to specify this in a consistent manner. It is usual to 
decompose criteria to a level of detail which allows this. 

(4) Non-redundancy: Is there more than one criterion measuring the same factor? When eliciting 
ideas often the same concept may arise under different headings. One can easily check for criteria 
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which appear to be measuring the same thing by calculating a correlation coefficient if appropriate 
data is available, or carrying out a process of matching as associated with analysis of repertory grids. 

(5) Judgmental independence: Criteria are not judgementally independent if preferences with respect 
to a single criterion, or trade-offs between two criteria, depend on the level of another. 

(6) Balancing completeness and conciseness: a number of authors note that desirable characteristics 
of a value tree are that it is complete, i.e., that all important aspects of the problem are captured, and 
also that it is concise, keeping the level of detail to the minimum required. 

(7) Operationality: The model is usable with reasonable effort - that the information required does not 
place excessive demands on the decision makers. The context in which the model is being used is 
clearly important in judging the usability of a model. 

(8) Simplicity versus complexity: The value tree, or criteria set is itself a simple representation, 
capturing the essence of a problem, which has been extracted from a complex problem description. 
The modeller should strive for the simplest tree which adequately captures the problem for the 
decision maker. 

LOs quality evaluation model based on these MCDA criteria identification principles is presented in 
the Fig. 1. This model consists of eight quality criteria, four of them dealing with technological quality, 
three - with pedagogical quality of LOs, and one - with IPR issues. This model includes three groups 
of criteria, namely, technological, pedagogical and IPR criteria. 



Figure 1: LOs quality evaluation model (criteria tree) 
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According to the technological quality criteria classification principle, we can divide technological 
quality criteria into ‘internal quality’ and ‘quality in use’ criteria of the educational software (e.g., LOs). 
‘Internal quality’ is a descriptive characteristic that describes the quality of software independently 
from any particular context of its use, while ‘quality in use’ is evaluative characteristic of software 
obtained by making a judgment based on criteria that determine the worthiness of software for a 
particular project (Kurilovas and Dagiene 2009a). 

Any LOs quality evaluation model (set of criteria) should provide the experts (decision makers) the 
clear instrumentality who (i.e., what kind of experts) should analyse what kind of LOs quality criteria in 
order to select the best LOs suitable for their needs. According to aforementioned technological 
quality criteria classification principle, ‘internal quality’ criteria should be mainly the area of interest of 
the software engineers, and ‘quality in use’ criteria should be mostly analysed by the programmers 
and users taking into account the users’ feedback on the usability of software (Kurilovas and 
Serikoviene 2010). 

The authors have applied these two principles in their previous papers (Kurilovas and Dagiene 2009a, 
2009b, 2009c, Kurilovas and Serikoviene 2010) on technological evaluation of the learning software, 
and thus have identified a number of LOs technological quality evaluation criteria presented in the 
technological part of the LOs quality evaluation model (see Fig. 1). 

On the other hand, the authors have analysed a number of existing models (sets of quality evaluation 
criteria) for evaluation of pedagogical quality of LOs, e.g., (Becta 2007, Leacock and Nesbit 2007, 
MELT 2008, Vargo et. al. 2003). 

Suitable pedagogical reusability criteria based on MCDA principles (Belton and Stewart 2002) are: 

(1) Interactivity, strong visual structure (animations, images and short videos are travelling best). 

(2) Language independence or low language dependence (easily translatable) or multilinguality. 

(3) Ease of use, intuitiveness. 

Intellectual property rights (IPR) criterion should also be considered here (Kurilovas and Bireniene 
2010 ). 

The authors’ analysis has shown that the model presented in Fig. 1 fits all MCDA criteria identification 
principles. Taking into account Non-redundancy, Judgmental independence, Balancing completeness 
and conciseness, Operationality, and Simplicity versus complexity MCDA criteria identification 
principles, the authors consider that the following eight LOs evaluation criteria should construct the 
comprehensive LOs quality criteria tree (see Fig. 1): 

Technological quality criteria: 

‘Internal quality’ criteria: 

(1) Technological reusability: 

■ Interoperability: Metadata accuracy; Compliance with the main import/export standards (e.g., IMS 
CP, IMS CC, SCORM 2004). 

■ Decontextualisation: Is LO indivisible (atomic)? - LO aggregation (granularity) level; Is LO 
modular (i.e., are the parts of a content item fully functional on their own?). 

■ Cultural and learning diversity (Adaptability): LO flexibility (LO can be modified, for instance from 
a configuration file, from a plain text file or because it is provided along with its source code or an 
authoring tool); LO internationalisation level; LO suitability for localisation. 

« Accessibility (design of controls and presentation formats to accommodate disabled and mobile 
learners): Is LO designed for all?; Compliance with accessibility standards (W3C) (Kurilovas and 
Dagiene 2009c). 

(2) Architecture: Is LO architecture layered in order to separate data, presentation and application 
logics? (Kurilovas and Dagiene 2009c). 
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(3) Robustness, technical stability: 

■ Having help functions that identify common user problems and their solutions. 

■ Having navigational actions that can be undone. 

■ Giving quick, visible and audible responses to user actions. 

■ Allowing the user to exit at any point. 

■ Not being adversely affected by user experimentation and error. If users do experience an error 
they should be able to recover quickly and, where appropriate, be informed about the nature of 
the error (Becta, 2007) 

‘Quality in use’ criterion: 

(4) Design and usability (design of visual and auditory information for enhanced learning and efficient 
mental processing): Aesthetics; Navigation; User-friendly interface; Information structuring; 
Personalisation (Kurilovas and Dagiene 2009c). 

These LOs technological quality criteria are included into a majority of the aforementioned LOs 
evaluation models. ‘Interoperability’ and ‘Accessibility’ criteria being independent criteria in e.g., 
(Leacock and Nesbit 2007) or (Becta 2007) are included as sub-criteria into Technical reusability’ 
criterion in the presented model. There are several reasons for this, e.g., both ‘Interoperability’ and 
‘Accessibility’ criteria deal with international interoperability standards and specifications, both 
influence LO technical reusability level in different repositories and platforms, etc. MCDA Non¬ 
redundancy principle is applied here. 

Pedagogical guality criteria: 

(5) Interactivity, strong visual element: e.g., LOs include animations, images, short videos and 
simulations that are self-explanatory or have just a few text labels or icons/buttons for start, stop, etc.; 
strong visual structure (MELT 2008). 

(6) Language independence: LO is not text-heavy; LOs may have little or no text; or low language 
dependence (easily translatable); or LOs are multilingual, i.e., LOs have been designed to be 
language customisable and are already offered in more than one language (MELT 2008). 

(7) Ease of use, intuitiveness: 

■ Users can find their way through the resource almost intuitively; they can broadly understand 
what is the intended learning objective or topic (MELT 2008). 

■ LOs provide appropriate guidance, where necessary, for learners and/or practitioners. 

■ LOs make appropriate assumptions about the ICT skills of users, both learners and practitioners, 
or provide straightforward guidance on this. 

■ LOs not present a barrier or impede the learning experience (Becta 2007). 

These LOs pedagogical quality criteria are included into the analysed LOs evaluation models 
(Leacock and Nesbit 2007, Becta 2007, MELT 2008, Vargo et. al. 2003). 

There are also several criteria included into the aforementioned models. They are: ‘Content quality’ 
(Leacock and Nesbit 2007), ‘Match to the curriculum’, ‘Assessment’, ‘Learner engagement’, and 
‘Innovative approaches’ (Becta 2007). 

In the authors’ opinion, ‘Content quality’ criterion should not be included into the proposed LOs quality 
evaluation model. The main reason for this is “a need for a common (more narrow) definition of what 
is, and what is not a LO” (Paulsson and Naeve 2006). Therefore, since a LO is “any digital resource 
that can be reused to support learning” (Wiley 2000), we can conclude that scientific content of any 
LO should be relevant, accurate and trustworthy, in other case the digital resource could not be used 
and reused “to support learning”. 

‘Match to the curriculum’ criterion could be suitable for nationally recognised quality criteria of learning 
resources, but taking into account the “more narrow definition” (Paulsson and Naeve 2006) of LOs, 
we should consider only the resources “that can be reused” to support learning (Wiley 2000). Since 
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there are different curricula in different countries, reusable resources should be curriculum 
independent. Reusable LOs could be often used in other pedagogic situations and learning scenarios 
that have been planned by LOs authors. Such approach has been, e.g., applied by the author’s 
leaded team in FP6 CALIBRATE project (CALIBRATE 2008) where Lithuanian teachers have been 
created and implemented their own lesson plans using foreign LOs from LRE. 

Criteria such as ‘Assessment to support learning’ and ‘Robust summative assessment’ (Becta 2007) 
often do not fit the LO reusability principle - they are mostly suitable for entire learning courses with 
high semantic density and aggregation level. 

Criteria such as ‘Learner engagement’ and ‘Innovative approaches' are interconnected on the one 
hand, and are closely related to ‘Interactivity’ and ‘Intuitiveness’ criteria included into the proposed 
model, on the other. 


IPR quality criterion: 

(8) Open license, free to use, open code: Licensing (clear rules, e.g., compliance with Creative 
Commons); Economic efficiency - Cost versus Ouality taking into account probable LO reusability 
level (Kurilovas 2007). 

3.2 Learning objects quality evaluation methods 

The widely used measurement criteria of the decision attributes’ quality are mainly qualitative and 
subjective. Decisions in this context are often expressed in natural language, and evaluators are 
unable to assign exact numerical values to the different criteria. Assessment can be often performed 
by linguistic variables: ‘bad’, ‘poor’, ‘fair’, ‘good’ and ‘excellent’. These values are imprecise and 
uncertain: they are commonly called ‘fuzzy values’. Integrating these different judgments to obtain a 
final evaluation is not evident (Kurilovas and Serikoviene 2010). Therefore, the authors have 
proposed to use fuzzy group decision making theory (Ounaies et. al. 2009) to obtain final assessment 
measures. The fuzzy numbers are: (1) triangular fuzzy numbers, (2) trapezoidal fuzzy numbers, and 
(3) bell-shaped fuzzy numbers. In the presented paper, the authors use triangular and trapezoidal 
fuzzy numbers for evaluating quality and reusability of LOs. 


Use of triangular fuzzy numbers 


According to (Zhang Li Li and Cheng De Yong 1992), triangular fuzzy numbers (TFNs) are a class of 
the fuzzy set representation. A triangular fuzzy number is expressed by three real numbers M = (I, m, 
u); the parameters I, m and u, respectively, indicate the lower, the mean and the upper possible 
values. TFNs membership functions are as follows: 


Pm(.x) 


x-l 

m — V 
< x —u 

m — u’ 
V 0, 


if x E [l, m], 

if x E [m,u], 
if x E [/,«]. 


Figure 2 illustrates triangular fuzzy numbers. 



Figure 2: T riangular fuzzy numbers 
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Conversion of these qualitative values into fuzzy numbers is shown in Table 1. 


Table 1 : Linguistic variables conversion into triangular fuzzy numbers 


Linguistic variables 

Triangular fuzzy numbers 

Excellent 

(0.700, 0.850, 1.000) 

Good 

(0.525, 0.675, 0.825) 

Fair 

(0.350, 0.500, 0.650) 

Poor 

(0.175, 0.325, 0.475) 

Bad 

(0.000, 0.150, 0.300) 


Therefore, in the case of using average triangular fuzzy numbers, linguistic variables conversion into 
non-fuzzy values of the evaluation criteria should be as follows: ‘excellent’=0.850; ‘good‘=0.675; 
‘fair’=0.500; ‘poor’=0.325; ‘bad’=0.150 (Kurilovas and Serikoviene 2010). 


The weight of the evaluation criterion reflects the experts’ opinion on the criterion’s importance level in 
comparison with the other criteria for the particular needs. For example, for the most simple (general) 
case, when all LOs evaluation criteria are of equal importance (i.e., we pay no especial attention to 
LOs reusability criteria), the experts could consider the equal weights a, = 0.125 according to the 
normalisation requirement (2). 

But if we pay especial attention to LOs reusability criteria, we can, e.g., consider the increased 
weights for the 1 st and 6 th LOs quality evaluation criteria (see Fig. 1 and Tables 2, 3 below), because 
these criteria deal with LOs reusability mostly. In this case, the authors while implementing eQNet 
have decided to apply increased weights that are twice higher in comparison with the other ones (i.e., 
0.2), and all other criteria weights according to normalisation requirement (2) should be equal 0.1. 

Lithuanian Physics expert teacher (the co-author of the paper) has applied the presented evaluation 
model and triangular fuzzy numbers method in eQNet project (see Table 2 below). 

A number of probably qualitative reusable Physics LOs have been identified in Lithuanian LOs 
repositories and evaluated against the model and method presented above (see formula (1)). 

There are several examples of these LOs presented in Table 2: 

(1) LOi: Light diffraction (available online at http://mkp.emokvkla.lt/imo/lt/mo/330/ ): 

(2) L0 2 : Photo effect (available online at http://mkp.emokykla.lt/imo/lt/mo/367/ ): and 


(3) L0 3 : Isobar process (available online at http://mkp.emokykla.lt/imo/lt/mo/395/) . 

Table 2: Results of experimental evaluation of Physics LOs general quality (q) and “travel well” 
quality (twq) using triangular fuzzy numbers method 


LOs evaluation criteria 

LOiq 

L0 2 q 

L0 3 q 

LOi twq 

L0 2 twq 

L0 3 twq 

Technological criteria: 







1. Technological reusability 

0.850 

0.850 

0.850 

0.1700 

0.1700 

0.1700 

2. Design and usability 

0.850 

0.850 

0.850 

0.0850 

0.0850 

0.0850 

3. Working stability 

0.850 

0.850 

0.850 

0.0850 

0.0850 

0.0850 

4. Architecture 

0.675 

0.675 

0.675 

0.0675 

0.0675 

0.0675 

Pedagogical criteria: 







5. Interactivity level 

0.500 

0.500 

0.500 

0.0500 

0.0500 

0.0500 

6. Language independence 

0.850 

0.675 

0.500 

0.1700 

0.1350 

0.1000 

7. Ease of use, intuitiveness 

0.850 

0.850 

0.850 

0.0850 

0.0850 

0.0850 

IPR criteria: 







8. Open licence, cost 

0.850 

0.850 

0.850 

0.0850 

0.0850 

0.0850 

Evaluation results: 

0.7844 

0.7625 

0.7406 

0.7975 

0.7625 

0.7275 
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These results mean that LOi meets 78.44% quality (q) in comparison with the ideal, L0 2 - 76.25%, 
and L0 3 - 74.06%. They also mean that LO-i meets 79.75% “travel well” quality (twq) in comparison 
with the ideal, L0 2 - 76.25%, and l_0 3 - 72.75%. 

Therefore, using triangular fuzzy numbers method, one could see that LOi is the best alternative 
(among the evaluated) both from general quality and “travel well” quality points of view. 

Lithuanian Mathematics expert teacher (the co-author of the paper) has also applied the presented 
evaluation model and triangular fuzzy numbers method in eQNet project (see Table 3). 

A number of probably qualitative reusable LOs have been identified in Lithuanian LOs repositories 
and evaluated against the aforementioned model and method (see formula (1)). 

There are three examples of these LOs presented in Table 3: 

* LO-p “Coordinate Method” (available online at http://mkp.emokykla.lt/imo/lt/mo/250/ ); 

■ L0 2 : “Polygon area” (available online at http://mkp.emokvkla.lt/imo/lt/mo/431/) : and 

■ L0 3 : “Interval Method” (available online at http://mkp.emokvkla.lt/imo/lt/mo/316/ ). 

Table 3: Results of experimental evaluation of Mathematics LOs general quality (q) and “travel well” 
quality (twq) using triangular fuzzy numbers method 


LOs evaluation criteria 

LOiq 

L0 2 q 

L0 3 q 

LOi twq 

L0 2 twq 

L0 3 twq 

Technological criteria: 







1. Technological reusability 

0.675 

0.850 

0.675 

0.1350 

0.1700 

0.1350 

2. Design and usability 

0.675 

0.850 

0.850 

0.0675 

0.0850 

0.0850 

3. Working stability 

0.675 

0.500 

0.675 

0.0675 

0.0500 

0.0675 

4. Architecture 

0.675 

0.500 

0.500 

0.0675 

0.0500 

0.0500 

Pedagogical criteria: 







5. Interactivity level 

0.850 

0.500 

0.325 

0.0850 

0.0500 

0.0325 

6. Language independence 

0.675 

0.850 

0.325 

0.1350 

0.1700 

0.0650 

7. Ease of use, intuitiveness 

0.850 

0.850 

0.500 

0.0850 

0.0850 

0.0500 

IPR criteria: 







8. Open licence, cost 

0.850 

0.850 

0.850 

0.0850 

0.0850 

0.0850 

Evaluation results: 

0.7406 

0.7188 

0.5875 

0.7275 

0.7450 

0.5700 


These results mean that LO-i meets 74.06% general quality (q) in comparison with the ideal, L0 2 - 
71.88%, and L0 3 - 58.75%. They also mean that LOi meets 72.75% “travel well” quality (twq) in 
comparison with the ideal, L0 2 - 74.50%, and L0 3 - 57.00%. 


Therefore, using triangular fuzzy numbers method, one could see that LOi is the best alternative 
(among the evaluated) from general quality point of view, but L0 2 is the best from “travel well” quality 
point of view. 

These two examples show that the application of the presented model and triangular fuzzy numbers 
method for evaluation of LOs quality can show both 

■ Similar results for ‘general quality’ and ‘travel well quality’ of LOs (see Table 2), and 

■ Different results for ‘general quality’ and ‘travel well quality’ (see Table 3) of different LOs. 

Use of trapezoidal fuzzy numbers 

A trapezoidal fuzzy number is a fuzzy number represented by four points as follows: M = (a, b, c, d). 

In this case, a membership function can be attached to the level fuzzy function: 
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f 0 , if 

I x — a 

I,- */ 

' b-a 

M rW=. I f 

\d-x 

I;— f 

i a —c 
l 0, if 

Figure 3 illustrates trapezoidal fuzzy numbers. 



x <a, 
a<x<b, 
b <x<c, 
c < x <d, 
x > d. 


Figure 3: Trapezoidal fuzzy numbers 

Conversion of these qualitative values into fuzzy numbers is shown in Table 4. 
Table 4: Linguistic variables conversion into trapezoidal fuzzy numbers 


Linguistic variables 

Trapezoidal fuzzy numbers 

Excellent 

(0.800, 1.000, 1.000, 1.000) 

Good 

(0.600, 0.800, 0.800, 1.000) 

Fair 

(0.300, 0.500, 0.500, 0.700) 

Poor 

(0.000, 0.200, 0.200, 0.400) 

Bad 

(0.000, 0.000, 0.000, 0.200) 


Therefore, in the case of using secondary trapezoidal fuzzy numbers, linguistic variables conversion 
into non-fuzzy values of the evaluation criteria should be as follows: 'excellent’=1.000; ‘good -0.800; 
‘fair’=0.500; ‘poor’=0.200; ‘bad-0.000. The results of evaluation of quality of the same Physics and 
Mathematics LOs identified in Lithuanian LOs repositories using trapezoidal fuzzy numbers and the 
same approach to the weights of criteria are presented in Tables 5 and 6 below. 


Table 5: Results of experimental evaluation of Physics LOs general quality (q) and “travel well” quality 
(twq) using trapezoidal fuzzy numbers 


LOs evaluation criteria 

LOiq 

L0 2 q 

L0 3 q 

LOitwq 

L0 2 twq 

L0 3 twq 

Technological criteria: 







1. Technological reusability 

1.000 

1.000 

1.000 

0.2000 

0.2000 

0.2000 

2. Design and usability 

1.000 

1.000 

1.000 

0.1000 

0.1000 

0.1000 

3. Working stability 

1.000 

1.000 

1.000 

0.1000 

0.1000 

0.1000 

4. Architecture 

0.800 

0.800 

0.800 

0.0800 

0.0800 

0.0800 

Pedagogical criteria: 







5. Interactivity level 

0.500 

0.500 

0.500 

0.0500 

0.0500 

0.0500 

6. Language independence 

1.000 

0.800 

0.500 

0.2000 

0.1600 

0.1000 

7. Ease of use, intuitiveness 

1.000 

1.000 

1.000 

0.1000 

0.1000 

0.1000 

IPR criteria: 







8. Open licence, cost 

1.000 

1.000 

1.000 

0.1000 

0.1000 

0.1000 

Evaluation results: 

0.9125 

0.8875 

0.8500 

0.9300 

0.8900 

0.8300 
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These results mean that LC^ meets 91.25% quality (q) in comparison with the ideal, L0 2 - 88.75%, 
and L0 3 - 85.00%. They also mean that LO-i meets 93.00% “travel well” quality (twq) in comparison 
with the ideal, L0 2 - 89.00%, and L0 3 - 83.00%. 

Therefore, using trapezoidal fuzzy numbers method, one could see that LO-i is the best alternative 
(among the evaluated) both from general quality and “travel well” quality points of view. The results 
are similar in comparison with the ones obtained using triangular fuzzy numbers. 


Table 6: Results of experimental evaluation of Mathematics LOs general quality (q) and “travel well” 
quality (twq) using trapezoidal fuzzy numbers 


LOs evaluation criteria 

LO iq 

L0 2 q 

L0 3 q 

LOitwq 

L0 2 twq 

L0 3 twq 

Technological criteria: 







1. Technological reusability 

0.800 

1.000 

0.800 

0.1600 

0.2000 

0.1600 

2. Design and usability 

0.800 

1.000 

1.000 

0.0800 

0.1000 

0.1000 

3. Working stability 

0.800 

0.500 

0.800 

0.0800 

0.0500 

0.0800 

4. Architecture 

0.800 

0.500 

0.500 

0.0800 

0.0500 

0.0500 

Pedagogical criteria: 







5. Interactivity level 

1.000 

0.500 

0.200 

0.1000 

0.0500 

0.0200 

6. Language independence 

0.800 

1.000 

0.200 

0.1600 

0.2000 

0.0400 

7. Ease of use, intuitiveness 

1.000 

1.000 

0.500 

0.1000 

0.1000 

0.0500 

IPR criteria: 







8. Open licence, cost 

1.000 

1.000 

1.000 

0.1000 

0.1000 

0.1000 

Evaluation results: 

0.8750 

0.8125 

0.6250 

0.8600 

0.8500 

0.6000 


These results mean that LO-i meets 87.50% general quality (q) in comparison with the ideal, L0 2 - 
81.25%, and L0 3 - 62.50%. They also mean that LCL meets 86.00% “travel well” quality (twq) in 
comparison with the ideal, L0 2 - 85.00%, and L0 3 - 60.00%. 


Therefore, using trapezoidal fuzzy numbers method, one could see that LO-i is the best alternative 
(among the evaluated) both from general quality and “travel well” quality points of view. The general 
quality evaluation results are similar, but “travel well” quality evaluation results are a little bit different 
in comparison with the ones obtained using triangular fuzzy numbers. 

In real life situations a teacher is the only suitable expert to decide on quality of LOs, and, therefore, 
on purposefulness to use these LOs in his / her teaching process in particular school. 

In the analysed cases it is clear that Physics teacher should choose LO-i as the best alternative both 
from general quality and “travel well” quality points of view. 

Mathematics teacher should choose LO-i as the best alternative from general quality point of view, 
and either LO-i or L0 2 - from “travel well” quality point of view since the difference between “travel 
well” quality evaluation values of LO-i and L0 2 is very small. 

4. Conclusion and recommendations 

The research results presented in the paper show that the complex application of the principles of 
multiple criteria decision analysis for identification of quality evaluation criteria, technological quality 
criteria classification principle, fuzzy group decision making theory to obtain final evaluation 
measures, and normalisation requirement for the weights of evaluation criteria, as well as triangular 
and trapezoidal fuzzy numbers methods for LOs quality optimisation are (1) applicable in real life 
situations when schools have to decide on purchase of LOs for their education needs, and (2) could 
significantly improve the quality of expert evaluation of LOs by noticeably reduce of the expert 
evaluation subjectivity level. 

The experimental evaluation results show that the proposed scientific approaches are quite objective, 
exact and simply to use for selecting the qualitative LOs alternatives in the market. 
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On the other hand, the proposed LOs “travel well” quality evaluation approach is applicable for the 
aims of eQNet project in order to select “travel well” LOs from LRE or elsewhere to use them in the 
other education contexts and countries. 

Therefore, these approaches have been recommended by the authors to be widely used by European 
policy makers, publishers, practitioners, and experts-evaluators both inside and outside eQNet project 
to evaluate quality and reusability level of LOs. 

5. Appendix 

The work presented in this paper is partially supported by the European Commission under the 
LifeLong Learning programme - as part of the eQNet project (project number 502857-LLP-1-2009-1- 
BECOMENIUS-CNW). The authors are solely responsible for the content of this paper. It does not 
represent the opinion of the European Commission, and the European Commission is not responsible 
for any use that might be made of data appearing therein. 

References 

Becta, (2007) “Quality principles for digital learning resources”, 2007 

Belton, V., Stewart, T.J., (2002) “Multiple criteria decision analysis: an integrated approach”. Kluwer Academic 
Publishers, 2002 

CALIBRATE, (2008): FP6 1ST CALIBRATE (Calibrating eLearning in Schools) project web site. 
http://calibrate.eun.org 

eQNet: Quality Network for a European Learning Resource Exchange project website, (2011) 
http://eqnet.eun.org 

ISO/IEC 14598-1:1999 (1999) “Information Technology- Software Product Evaluation - Part 1: General 
Overview”. First edition, 1999-04-15 

Kurilovas, E., (2009) “Interoperability, Standards and Metadata for e-Learning”. In: G.A. Papadopoulos and C. 
Badica (Eds.): Intelligent Distributed Computing III, Studies in Computational Intelligence 237, pp. 121-130. 
Springer-Verlag Berlin Heidelberg 2009 

Kurilovas, E., (2007) “Digital Library of Educational Resources and Services: Evaluation of Components”. 

Informacijos mokslai. Vilnius, 2007, Vol. 42-43, pp. 69-77 
Kurilovas, E., Bireniene, V., (2010) “Application of Multiple Criteria Decision Analysis Model and Method for 

Evaluation of Quality of Learning Objects for Physics Subject in eQNet Project”. In: Proceedings of the 16 th 
National Scientific Practical Conference “Natural Science Education at a General School - 2010”. Anyksciai, 
Lithuania, 23-24 April, 2010, pp. 84-89 

Kurilovas, E., Dagiene, V., (2009) “Multiple Criteria Comparative Evaluation of e-Learning Systems and 
Components”. Informatica. 2009, Vol. 20, No. 4, pp. 499-518 
Kurilovas, E., Dagiene, V., (2009) “Quality Evaluation and Optimisation of e-Learning System Components”. In: 
Proceedings of the 8 th European Conference on e-Learning (ECEL’09). Bari, Italy, October 29-30, 2009, pp. 
315-324 

Kurilovas, E., Dagiene, V., (2009) “Learning Objects and Virtual Learning Environments Technical Evaluation 
Criteria”. Electronic Journal of e-Learning, 2009, Vol. 7, Issue 2, pp. 127-136 
Kurilovas, E., Serikoviene, S., (2010) “Learning Content and Software Evaluation and Personalisation Problems”. 
Informatics in Education, 2010, Vol. 9 (1), pp. 91-114 

Leacock, T. L., Nesbit, J. C., (2007) “A Framework for Evaluating the Quality of Multimedia Learning Resources”. 

Educational Technology & Society, 2007, 10 (2), pp. 44-59 
LRE, (2011): European Learning Resource Exchange service for schools web site. 

http://lreforschools.eun.org/LRE-Portal/lndex.iface 
McCormick, R., Scrimshaw, P., Li, N., and Clifford, C. (2004). CELEBRATE Evaluation report. 

http://celebrate.eun.org/eun.org2/eun/lnclude_to_content/celebrate/file/Deliverable7_2EvaluationReport02D 

ec04.pdf 

MELT, (2008): EU eContentp/us programme’s ‘Metadata Ecology for Learning and Teaching’ project web site. 
http://melt-project.eun.org 

Paulsson, F., Naeve, A., (2006) “Establishing technical quality criteria for Learning Objects”. 

http://www.frepa.org/wp/wp-content/files/Paulsson-Establ-Tech-Qual_finalv1.pdf 
Oliver, M., (2000) “An introduction to the Evaluation of Learning Technology”. Educational Technology & Society 
3(4): 20-30. 

Ounaies, H.Z., Jamoussi, Y., Ben Ghezala, H.H., (2009) “Evaluation framework based on fuzzy measured 
method in adaptive learning system”. Themes in Science and Technology Education, 2009, Vol. 1, Nr. 1, 
pp.49-58 

Vargo, J., Nesbit, J. C., Belfer, K., Archambault, A., (2003) “Learning object evaluation: Computer mediated 
collaboration and inter-rater reliability”. International Journal of Computers and Applications, 2003, Vol. 25 
(3), pp 198-205 

Wiley. D. A., (2000) “Connecting Learning Objects to Instructional design Theory: a definition, a Metaphor, and a 
Taxonomy”. Utah State University, http://www.reusability.org/read/ 


www.eiel.org 


50 


©Academic Publishing International Ltd 



Eugenijus Kurilovas et al 


Zavadskas, E. K., Turskis, Z., (2010) “A new additive ratio assessment (ARAS) method in multicriteria decision¬ 
making”. Technological and Economic Development of Economy 16(2): 159-172 
Zhang Li Li, Cheng De Yong, (1992) “Extent analysis and synthetic decision”. In: Support Systems for Decision 
and Negotiation Processes, Preprints of the /FAC/IFORS/IIASA/TIMS Workshop, Vol. 2. System Research 
Institute, Warsaw, pp. 633-640 


www.eiel orq 


51 


ISSN 1479-4403 



